Speech-enabled devices may provide audible answers to spoken utterances from users. Such devices are often the front-end for so-called digital assistant software or “digital assistants” or “virtual assistants”. Such devices often provide a limited amount of speech recognition functionality themselves, and communicate with a server or other device for additional resources. Further, such devices often receive data from the server or other device (e.g., in response to a voice-entered query or utterance) that is converted into audible synthesized speech for the user. For example, a user may say “Will it rain today?” and a speech-enabled device may audibly respond, “It will be sunny all day.” A user may provide an utterance that relates to the user's personal information. For example, a user might ask the system “What is on my calendar” and the device may respond with appointments on a calendar associated with the device. However, common speech enabled devices do not adequately distinguish between various users, especially as related to secure access to a specific user's personal information, while still permitting general use of the speech enabled device by, for example, guest users.